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IN THE CLAIMS : 

Please AMEND claims 10, 15, and 16, as follows. 

l.to9. (Cancelled). 

10. (Currently Amended) An image processing apparatus for compositing 
composting an image of a virtual object and an image of a physical space to generate a mixed 
reality image and causing a HMD to display the mixed reality image, comprising: 

a database which holds 3D CG data used for generating the image of the vii ' tual 
obiee t and scene graph data, wherein the 3D CG rendering data contains data representing 
geographical shape and color of the virtual object, texture data, and data representing the position 
and orientation of the virtual object, and wherein the scene graph data is data in which the 3D CG 
rendering data is hierarchically managed on the basis of the parent-child relationship between 
components of the virtual object : 

an image capturing unit which is attached to the HMD and captures the image 
of the physical space; 

a first measurement unit which measures a position and orientation of the 

HMD; 

an object manipulation unit which is used by a user wearing the HMD in order 
to operate a position and orientation of the virtual object; 

a second measurement unit which measures a position and orientation of said 
object manipulation unit; 
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an operation panel which can be operated by the user, arranged at a position in 
the physical space within a viewing field of said image capturing unit attached to the HMD, 
displays an operation panel image which is used for editing the virtual object and contains a 
region for hierarchically displaying information about each of the components in accordance with 
the scene graph data held in the database , and is capable of receiving a user instruction of editing 
the virtual object input by the user; 

an operation panel image generation unit which generates the operation panel 
image by using the data held in said database, and outputs the generated operation panel image to 
said operation panel; 

a rendering unit which updates the data held in said database according to the 
user instruction received via said operation panel and the measurement resuh of said second 
measurement unit, and renders, by using the updated data, the image of the virtual object 
according to the measurement results of said first and second measurement units; 

a composition unit which composites the image of the virtual object rendered 
by said rendering unit and the image of the physical space captured by said image capturing unit 
to generate the mixed reality image; and 

a HMD , to which said image capturing unit is attached, which displays the 
mixed reality image generated by said composition unit. 

11. (Previously Presented) The apparatus according to claim 10, wherein the 
image of the virtual object is generated on the basis of 3D CAD data of the virtual object, and 
said operation panel displays an assembly tree based on the 3D CAD data. 
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12. (Previously Presented) The apparatus according to claim 11, wherein a part, 
which is obtained by enlarging a designated part of the assembly tree included in the operation 
panel image, includes a component name contained in the assembly tree. 

13. (Previously Presented) The apparatus according to claim 10, wherein said 

operation panel includes a display device and an operation device, wherein the display device 
displays the operation panel image, and wherein the operation device is used for inputting the 
user instruction. 

14. (Previously Presented) The apparatus according to claim 10, wherein the 
HMD can present the mixed reality image to two eyes of a user who wears the HMD. 

15. (Currently Amended) An image processing method of compositing 
composting an image of a virtual object and an image of a physical space to generate a mixed 
reality image and causing a HMD to display the mixed reality image, comprising the steps of: 

holding 3D CG rendering data used for generating the image of the virtual 
object in a database and scene graph data, wherein the 3D CG rendering data contains data 
representing geographical shape and color of the virtual object, texture data, and data 
representing the position and orientation of the virtual object, and wherein the scene graph data is 
data in which the 3D CG rendering data is hierarchically managed on the basis of the parent-child 
relationship between components of the virtual object : 

providing an image capturing unit, which is attached to the HMD, to capture 
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the image of the physical space; 

measuring a position and orientation of the HMD with a first measurement 

unit; 

operating an object manipulation unit, by a user wearing the HMD, in order to 
position and orient the virtual object; 

measuring a position and orientation of the object manipulation unit with a 
second measurement unit; 

receiving a user instruction of editing the virtual object, input by the user using 
an operation panel, wherein the operation panel is arranged at a position in the physical space 
within a viewing field of the image capturing unit attached to the HMD and displays an operation 
panel image which is used for editing the virtual object , and contains a region for hierarchically 
displaying information about each of the components in accordance with the scene graph data 
held in the database : 

generating the operation panel image by using the data held in the database 
with an operation panel image generation unit and outputting the generated operation panel 
image to the operation panel; 

updating the data held in the database according to the user instruction 
received via the operation panel and the measurement result of the second measurement unit, and 
rendering, by using the updated data, the image of the virtual object according to the 
measurement results of the first and second measurement units; 

compositing the rendered image of the virtual object and the captured image of 
the physical space to generate the mixed reality image; and 
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displaying the generated mixed reality image on the HMD , to which the image 
capturing unit is attached . 

16. (Currently Amended) A computer-readable storage medium encoded with a 
computer program for an image processing method of compositing composting an image of a 
virtual object and an image of a physical space to generate a mixed reality image and causing a 
HMD to display the mixed reality image, comprising the steps of: 

holding 3D CG rendering data used for generating the image of the virtual 
ob i ect in a database and scene graph data, wherein the 3D CG rendering data contains data 
representing geographical shape and color of the virtual object, texture data, and data 
representing the position and orientation of the virtual object, and wherein the scene graph data is 
data in which the 3D CG rendering data is hierarchical managed on the basis of the parent-child 
relationship between components of the virtual object : 

providing an image capturing unit, which is attached to the HMD, to capture 
the image of the physical space; 

measuring a position and orientation of the HMD with a first measurement 

unit; 

operating an object manipulation unit, by a user wearing the HMD, in order to 
position and orient the virtual object; 

measuring a position and orientation of the object manipulation unit with a 
second measurement unit; 

receiving a user instruction of editing the virtual object, input by the user using 
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an operation panel, wherein the operation panel is arranged at a position in the physical space 
within a viewing field of the image capturing unit attached to the HMD and displays an operation 
panel image which is used for editing the virtual objec t, and contains a region for hierarchically 
displaying information about each of the components in accordance with the scene graph data 
held in the database : 

generating the operation panel image by using the data held in the database 
with an operation panel image generation unit and outputting the generated operation panel 
image to the operation panel arranged at the position in the physical space within the viewing 
field of said image capturing unit : 

updating the data held in the database according to the user instruction received 
via the operation panel and the measurement result of the second measurement unit, and 
rendering, by using the updated data, the image of the virtual object according to the 
measurement results of the first and second measurement units; 

compositing the rendered image of the virtual object and the captured image of 
the physical space to generate the mixed reality image; and 

displaying the generated mixed reality image on the HMD , to which the image 
capturing unit is attached . 
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